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IMAGE PROCESSOR AND MONITORING SYSTEM 



BACKGROUND OF THE INVENTION 



The present invention relates to a technique of generat- 



5 ing a synthesized image from multiple images that have been 
captured by cameras mounted on a vehicle , and more particu- 
larly relates to a technique effectively applicable to a 
monitoring system used as an safety check aid during vehicle 
Iq driving. 

in 10 A vehicle video monitoring system disclosed in Japanese //OS 



Laid-Open Publication No. 11-78692 is one of the known systems^/"' 
for monitoring the surroundings of a running vehicle using 
cameras. In this system, a vehicle is equipped with multiple 
cameras, which are mounted here and there on the vehicle body 

15 to take a video of its surroundings. The respective images 
taken by these cameras are transformed and synthesized to- 
gether while maintaining boundary continuity between them, 
thereby making a newly synthesized image and displaying it on 
the monitor screen. 

20 A vehicle surrounding monitoring system of this type, 

however, will normally carry out the image processing on the 
supposition that each camera once mounted should be unable to 
change its position or the direction it faces. This is be- 
cause it is practically impossible, or at least extremely 

25 difficult, to process multiple images for a synthesized image 
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in real time with the continuously changing positions and di- 
rections of the cameras taken into account. The reason is 
that it takes too much time to perform that complicated proc- 
essing and that the size and ability of such a system are 
5 limited. 

On the other hand, considering the aerodynamics and des- 
ign of a vehicle, it is highly probable that those cameras 
for use in such a monitoring system are mounted on various 
movable parts of the vehicle. For example, those cameras may 
10 be mounted around the rearview mirrors or on the trunk or 
hood of the vehicle. 

Then, the following problems arise. 

Specifically, when any of those movable parts, on which 
the camera is mounted, changes its state (e.g., moves), the 

15 image taken by that camera is affected by the change to pos- 
sibly misalign the image that forms an integral part of the 
resultant synthesized image. For example, where a camera is 
mounted around an outside rearview mirror, the opening of a 
door with the mirror changes the direction the camera faces . 

20 As a result, only a portion of the synthesized image, corre- 
sponding to the image taken by the camera in question, mis- 
aligns, thus making the image on the screen unnatural to the 
viewer. 

In addition, even if one of those cameras is mounted on 
25 a fixed part of the vehicle, a similar phenomenon is ob- 
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served. For example, when a particular camera takes a video 
of an opening door, only a portion of the synthesized image, 
corresponding to the image taken by the camera, also mis- 
aligns. Such partial misalignment makes the synthesized im- 
5 age unnatural to the viewer and should be eliminated - 

SUMMARY OF THE INVENTION 

An object of the invention is to make a synthesized im- 
age, generated by an image processor from multiple images ta- 

10 ken by cameras on a vehicle, as natural to a viewer as 
possible even if any of those images is affected by a state 
change of a movable part of the vehicle. 

Specifically, an inventive image processor includes an 
image processing section, which receives multiple images that 

15 have been taken by cameras mounted on a vehicle to monitor 
surroundings of the vehicle, generates a synthesized image 
from the images and then output the synthesized image to a 
display device. The image processing section switches dis- 
play modes of the synthesized image in accordance with a 

20 state of a movable part of the vehicle. 

In the present invention, the display modes of a synthe- 
sized image are switched in accordance with a state of a mov- 
able part of a vehicle. Accordingly, even if the synthesized 
image has become partially unnatural to a viewer due to a 

25 change in the state of the vehicle movable part, the unnatu- 
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ral image will not be displayed. Also, by switching the dis- 
play modes of the synthesized image, it is possible to let 
the user know quickly and timely that the movable part is now 
in a state unfavorable to the vehicle's running, for example. 
5 In one embodiment of the present invention, if a change 

in the state of the movable part has altered the position or 
direction of at least one of the cameras to make the synthe- 
sized image unnatural, the image processing section preferably 
switches the display modes of the synthesized image from a 

10 normal mode into an alert mode. 

In this particular embodiment, the image processing sec- 
tion may generate the synthesized image in the alert mode 
without using the image taken by the camera that has had its 
position or direction changed. 

15 Alternatively, the image processing section may output 

no synthesized image in the alert mode. 

As another alternative, the image processing section may 
also output an alert message instead of, or along with, the 
synthesized image in the alert mode. Then, the user can 

20 quickly sense the unnaturalness of the synthesized image 
thanks to the alert message. Also, where the alert message 
displayed indicates exactly how the movable part has changed 
its state, the user can know the current situation of the ve- 
hicle quickly and timely. 

25 In another embodiment of the present invention, if, due 
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to a change in the state of the movable part, the image of the 
movable part occupies a different portion of the image taken 
by at least one of the cameras to make the synthesized image 
unnatural, the image processing section preferably switches 
5 the display modes of the synthesized image from a normal mode 
into an alert mode. 

In this particular embodiment, the image processing sec- 
tion may generate the synthesized image in the alert mode 
without using the image which has been taken by the camera and 
10 the different portion of which the image of the movable part 
now occupies . 

Alternatively, the image processing section may output 
no synthesized image in the alert mode. 

As another alternative, the image processing section may 

15 output an alert message instead of, or along with, the syn- 
thesized image in the alert mode. Then, the user can quickly 
sense the unnaturalness of the synthesized image thanks to 
the alert message. Also, where the alert message displayed 
indicates exactly how the movable part has changed its State, 

20 the user can know the current situation of the vehicle quick- 
ly and timely. 

In still another embodiment, the movable part may in- 
clude at least doors, hood and trunk of the vehicle. In that 
case, the image processing section may switch the display 
25 modes of the synthesized image in accordance with opening and 
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closing of the doors, hood or trunk. 

In yet another embodiment, the movable part may include 
at least tires of the vehicle. In that case, the image proc- 
essing section may switch the display modes of the synthesized 
5 image in accordance with a steering angle of wheels on which 
the tires are fitted. 

A monitoring system according to the present invention 
includes the inventive image processor. The system includes: 
multiple cameras, mounted on a vehicle, for taking images of 

10 surroundings of the vehicle; a display device; and an image 
processing section. The image processing section receives 
the images taken by the cameras, generates a synthesized im- 
age from these images and then outputs the synthesized image 
to the display device. The image processing section switches 

15 display modes of the synthesized image in accordance with a 
state of a movable part of the vehicle. 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram illustrating a configuration 
20 for a monitoring system according to a first embodiment of 
the present invention* 

FIG. 2 illustrates an exemplary arrangement of cameras 
and exemplary images taken by the cameras according to the 
first embodiment. 
25 FIG. 3 illustrates an exemplary image, which has become 
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partially unnatural due to a change in the state of a movable 
part of a vehicle. 

FIGS . 4A through 4D illustrate how the display modes of 
a synthesized image may be switched in the present invention. 
5 FIGS. 5A through 5D illustrate examples of alert message 

images . 

FIG. 6 schematically illustrates an exemplary mapping 
table, 

FIG. 7 is a flowchart illustrating an overall image syn- 
10 thesis process. 

FIG. 8 is a flowchart illustrating the detailed process 
steps of Step S20 shown in FIG. 7. 

FIG. 9 is a flowchart illustrating the detailed process 
steps of Step S30 shown in FIG. 7. 
15 FIG. 10 is a block diagram illustrating a configuration 

for a monitoring system according to a second embodiment of 
the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

20 Hereinafter, preferred embodiments of the present inven- 

tion will be described with reference to the accompanying 
drawings . 

EMBODIMENT 1 

25 FIG. 1 is a block diagram illustrating a configuration 
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for a monitoring system according to a first embodiment of 
the present invention. In the monitoring system shown in 
FIG, 1, an image processing section 2 receives multiple cam- 
era images that an imaging section 1 has output, and combines 
5 these images together to generate a newly synthesized image, 
which will be presented on a display device 3. That is to 
say, the image processing section 2 is the core of the inven- 
tive image processor. 

The following Table 1 shows where the cameras may be 
10 mounted according to the first embodiment: 
[Table 1] 



Camera No . 


Position 


0 


Hood 


1 


Right door front 


2 


Right door rear 


3 


Trunk 


4 


Hood 


5 


Left door front 


6 


Left door rear 


7 


Trunk 



FIG. 2 illustrates this exemplary camera arrangement along 
with exemplary images taken by these cameras in this embodi- 
ment. In the illustrated embodiment, the eight cameras are 

15 mounted as shown in Table 1 and FIG. 2. When the camera ar- 
rangement shown in Table 1 and FIG. 2 is employed, the posi- 
tions and directions of the respective cameras are preferably 
determined so that the images taken by an adjacent pair of 
cameras share some image portion to eliminate blind spots for 

20 the viewer as much as possible. 
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The image processing section 2 transforms and combines 
together the eight camera images shown in FIG. 2, thereby gen- 
erating a synthesized image of the vehicle as viewed vertical- 
ly from over it, for example- To generate the synthesized 
5 image, the image transformation and synthesis of the trans- 
formed partial images (including boundary processing) are 
needed. In the system shown in FIG. 1, the image processing 
section 2 includes a mapping table searcher 22. And to proc- 
% ess those images by a single process step, the image process- 
'ft 10 ing section 2 uses a mapping table MPT, which will be 

□ described in detail later. 

U A mapping table selector 41 selects the type of a synthe- 

□ sized image to be displayed and specifies one of the mapping 
tables MPT in accordance with the type of the image in need. 

= 15 This selection may be carried out either manually or automati- 
cally depending on the condition of the vehicle (e.g., gear 
position, steering angle or the ON/OFF states of blinkers). A 
signal specifying the mapping table MPT to be selected is also 
output to an alert message selector 26 as well as to the map- 

20 ping table searcher 22. 

Responsive to a state signal SI indicating the current 
state of a vehicle movable part, a display mode changer 23 
switches the display modes of the synthesized image. For ex- 
ample, if a change in the state of the movable part has al- 

25 tered the position or direction of one of the cameras to make 




the synthesized image unnatural, the changer 23 switches the 
display modes of the synthesized image from a normal mode into 
an alert mode. In the alert mode, the synthesized image is 
generated without using the image taken by the camera in ques- 
5 tion that has had its position or direction changed. 

FIG. 3 illustrates an exemplary image, which has become 
partially unnatural due to a change in the state of a vehicle 
movable part. As can be seen from the synthesized image 
shown in FIG. 3, the images taken by cameras Nos. 1 and 2 

10 changed because the right door of the vehicle opened, and part 
of the synthesized image, corresponding to the images taken by 
these cameras Nos. 1 and 2, misaligned. 

Thus, in the illustrated embodiment, the opened/closed 
states of the right and left doors, trunk and hood, on which 

15 the cameras are mounted, are used as the states of vehicle 
movable parts. That is to say, the state signal SI, giving 
information "right door opened" or "hood closed", for example, 
is input to the display mode changer 23. Data about a corre- 
spondence between respective vehicle movable parts and the 

20 camera numbers is stored beforehand on the changer 23. The 
data may be as shown in the following Table 2, for example: 
[Table 2] 



Movable part 


Cameras mounted 


Right door 


Nos. 1 & 2 


Left door 


Nos. 5 & 6 


Trunk 


Nos. 3 & 7 


Hood 


Nos. 0 & 4 



In accordance with this correspondence and the information 
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given by the state signal SI, the changer 23 switches the 
display modes of the synthesized image. 

FIGS. 4A through 4D illustrate how the display modes of 
the synthesized image may be switched in the present inven- 
5 tion. Specifically, FIG. 4A illustrates a synthesized image 
displayed in normal mode. As shown in FIG. 4A, the movable 
parts of the vehicle are all closed and a natural image is 
presented. FIG. 4B illustrates an exemplary synthesized image 
displayed in alert mode. The image shown in FIG. 4B illus- 

10 trates a situation where the right door of the vehicle opened. 
In the synthesized image shown in FIG. 4B, the images taken by 
cameras Nos . 1 and 2, mounted on the right door of the vehi- 
cle, are not used for image synthesis and instead part of the 
synthesized image, corresponding to these images in question, 

15 is filled in a predetermined color. Alternatively, that part 
may be filled with any arbitrary pattern. Also, the colors 
may be changed at appropriate time intervals. 

In the illustrated embodiment, the display modes of the 
synthesized image are switched by rewriting the mapping data 

20 stored on the mapping table MPT. 

Optionally, in the alert mode, the output of the synthe- 
sized image itself may be suspended or the alert message se- 
lector 26 may output an alert message either alone or along 
with the synthesized image. 

25 FIG. 4C illustrates one such situation where no synthe- 
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sized image is displayed but only an alert message, indicating 
that the right door is open, is presented. On the other hand, 
FIG. 4D illustrates another such situation where the synthe- 
sized image shown in FIG. 4B is displayed along with the alert 
5 message shown in FIG. 4C. In this manner, camera images, af- 
fectable by any state change of a movable part, may be removed 
from the image synthesized and an alert message, informing the 
user of the state change of the movable part (e.g., a door is 
open), may be presented as shown in FIG. 4D. Then, the user 

10 (e.g., driver) of the vehicle can know the current state of 
his or her own car more quickly and more timely. 

The alert message selector 26 stores in advance those 
alert messages in image formats, for example. On receiving 
the state signal SI indicating the current state of a vehicle 

15 movable part and the signal output from the mapping table se- 
lector 41 to specify a mapping table MPT, the selector 26 se- 
lects one of the message images that has to be combined with 
the synthesized image. Then, the selector 26 sends out the 
message image to a video signal generator 24. 

20 FIGS. 5A through 5D illustrate examples of the alert 

message images stored on the selector 26. In the examples 
illustrated in FIGS. 5A through 5D, the message itself is dis- 
played in white (i.e., (255, 255, 255) where the image is pre- 
sented in RGB 256 gray scales), while the remaining part of 

25 the image is displayed in black (i.e., (0, 0, 0) in that 
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case). Each message image should have the same size as the 
synthesized image. 

The contents and display position of the message includ- 
ed in each message image may differ depending on which map- 
5 ping table was selected to generate the synthesized image and 
which movable part changed its state. Accordingly, various 
alert message images should preferably be prepared for all 
sorts of imaginable situations and one of those message images 
1% should preferably be selectable depending on the particular 
10 situation. 

□ The following Table 3 shows a correspondence among the 

i U mapping tables for use in image synthesis, various states of 
|3 the vehicle movable parts and the alert message images to be 
:2 displayed: 
:*f 15 [Table 3] 



Mapping Table 


State of movable part 


Message image No. 


Look-down 


R-Door open 


1 


Look-down 


L-Door open 


2 


Look-down 


Trunk open 


3 


Look-down 


Hood open 


4 


Blind corner monitor 


Hood open 


5 


Rearview monitor 


Trunk open 


6 


* 


All closed 


7 



For example, the second row of this Table 3 indicates that 
the mapping table now selected is for use to generate a "look- 
down" image and that if the right door, one of the vehicle's 
movable parts, is open, alert message image No. 1 should be 



20 selected. FIG. 5A illustrates an exemplary alert message ima- 
ge No. 1. In this situation, i.e., where the mapping table 
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currently selected will be used to generate a "look-down" im- 
age and the right door is open, the images taken by the cam- 
eras mounted on the right door will not be used for image 
synthesis. Accordingly, the alert message shown in FIG, 5A 
5 saying "R-Door Open" will be posted in part of the synthesized 
image where the images taken by the cameras mounted on the 
right door should be displayed. 

The alert message selector 26 stores beforehand informa- 
tion such as that shown in Table 3. Responsive to the state 

10 signal SI indicating the current state of the vehicle movable 
part and the signal specifying one of the mapping tables MPT, 
the selector 26 selects and outputs one of the message images 
to be posted. On receiving the message image from the selec- 
tor 26, the video signal generator 24 superimposes the message 

15 image on the synthesized image output from a pixel synthesizer 
21 and then displays the resultant combined image. 

The message image may be superimposed on the synthesized 
image by retrieving the RGB values of all pixels for the syn- 
thesized and message images and outputting the greater ones, 

20 for example. When this method is applied to the message im- 
age shown in FIG. 5A, the black pixel portions of the message 
image have RGB values smaller than those of the synthesized 
image in the associated portions. Accordingly, the RGB values 
of the synthesized image will be chosen. On the other hand, 

25 as for the white pixel portions of the message image, the RGB 
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values of the message image will always be chosen. In this 
manner, a combined image, in which only the message portions 
are superimposed on the synthesized image, can be obtained. 
In this case, part of the synthesized image where the camera 
5 images should not exist is preferably filled in a dark color 
to make the message superimposed in white as easily visible as 
possible. 

Also, if a combined image should be displayed by this 
5 superimposition method in a situation where all the movable 

fi 10 parts are closed (i.e., in the normal mode), the image filled 
□ in black (0, 0, 0) such as that shown in FIG. 5B may be used 

U as the message image. That is to say, the image shown in FIG. 

^ 5B may be prepared as message image No. 7 shown on the lower- 

if most row of Table 3. Naturally > the message selector 26 may 

*: 15 output no message image in the normal mode. 

Also, where a number of movable parts are open, the fol- 
lowing technique may be used. For example, if the mapping 
table now selected is for use to generate a "look-down" image 
and if the right and left doors are both open currently, then 
20 the message selector 26 selects the associated alert message 
images Nos. 1 and 2 shown in FIGS. 5A and 5C, respectively. 
Then, the message selector 26 retrieves the RGB values of all 
pixels for images Nos. 1 and 2 and chooses the greater values, 
thereby generating a newly combined message image shown in 
25 FIG. 5D, for example, and sending out the message image to the 
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video signal generator 24. 

In the foregoing illustrative embodiment, the alert mes- 
sage image contains alphabetical letters. Alternatively, the 
alert message may be a simplified diagram or illustration. 
5 Also, if no message images are to be displayed in the alert 
mode, then the alert message selector 26 may be naturally 
omitted . 

Hereinafter, it will be described in detail how to re- 
write the mapping data. 

10 As used herein, the "mapping table" means a table de- 

scribing the correspondence between each pixel included in a 
synthesized image and associated pixel data included in any of 
the camera images . The mapping table is used to speed up the 
synthesized image generation. 

15 There may be either one-to-one correspondence or multi- 

ple-to-one correspondence between pixel(s) included in each 
camera image and associated pixel included in the synthesized 
image. The type of correspondence is changeable depending on 
the position of a particular camera or the type of the synthe- 

20 sized image to be generated. For example, where boundary 
processing should be carried out for a boundary between multi- 
ple camera images, the multiple-to-one correspondence often 
has to be defined between pixels included in a plurality of 
camera images and an associated pixel in the synthesized im- 

25 age . 
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FIG. 6 schematically illustrates an exemplary mapping 
table. The table shown in FIG. 6 includes mapping data items 
corresponding to respective pairs of pixel coordinates (i, j) 
of the synthesized image. 
5 For example, a mapping data item MPl associated with pix- 

el coordinates (il, jl) is included in the mapping table. As 
can be seen from FIG. 6, this mapping data item MPl indicates 
that only the pixel data at the coordinates (12, 45) in the 
image taken by camera No. 1 is mapped to the pixel ( il , jl) of 

10 the synthesized image. This mapping table MPT further indi- 
cates a rate of necessity. For the mapping data item MPl, its 
rate of necessity is "1". So the pixel data at the coordi- 
nates (12, 45) in the image taken by camera No. 1 is used as 
it is as the pixel value at the coordinates (il, jl) of the 

15 synthesized image. 

Another mapping data item MP2 associated with pixel coor- 
dinates (i2, j2) is also included in this mapping table MPT. 
This mapping data item MP 2 indicates that pixel data at the 
coordinates (10, 10) in the image taken by camera No. 1 and 

20 another pixel data at the coordinates (56, 80) in the image 
taken by camera No. 0 are mapped to the pixel ( i2 , j2) of the 
synthesized image. Using the rates of necessity, the pixel 
value at the coordinates (i2, j2) in the synthesized image may 
be given by 

25 Pixel value = { (pixel value at the coordinates (56, 80) in 
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the image taken by camera No. 1) X 0.5+ (pixel value at 
the coordinates (10, 10) in the image taken by camera No. 
0) X0.3}/(0.5 + 0.3) 

By preparing mapping tables like this either automatical- 
5 ly through computations (e.g., geometric transformation) or 
manually, any desired synthesized image, representing a plan 
view, rearview or side view of a vehicle, can be generated 
quickly enough. Also, by changing the mapping tables for use 
to generate the synthesized images, the display modes of the 

10 synthesized image can be switched easily. This display mode 
switching may be carried out either manually or automatically 
in accordance with the state of a vehicle running. 

Specifically, the mapping tables may be stored on a ROM, 
including an electrically erasable and programmable ROM 

15 (EEPROM), or on a RAM. The mapping data may be stored there 
by writing the mapping data, computed by a microprocessor 
built in the image processing section, on the ROM or RAM, for 
example. Alternatively, mapping table data, available as 
firmware, may be written on the ROM or RAM by way of some data 

20 transfer means including telecommunications lines or disk 
drives . 

Next, it will be described exactly how to generate a 
synthesized image using the mapping table MPT and display mo- 
de changer 23. 

25 FIG. 7 is a flowchart illustrating an overall image syn- 
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thesis process according to this embodiment. First, in Step 
Sll, the imaging section 1 switches frame memories 12a, 12b, 
etc. A timing generator 25 generates a timing signal to pro- 
duce a sequence of moving pictures that will represent a syn- 
5 thesized image. Responsive to the timing signal output from 
the timing generator 25 to indicate the start of framing, the 
imaging section 1 switches the frame memories 12a, 12b, etc. 
That is to say, the imaging section 1 selects new combinations 
of frame memories on which the images taken by the cameras 11 

10 should be written and frame memories on which images to be 
referenced by the image processing section 2 should be stored. 
This switching operation is performed to avoid interference 
between writing and referencing because the pixels to be ref- 
erenced by the image processing section 2 are randomly located 

15 in various camera images irrespective of the order in which 
the images taken by the cameras 11 are written. 

Next, in Step S12, the timing generator 25 selects a pix- 
el to be processed from the synthesized image that will have 
to be output at the predetermined timing. 

20 Then, in Step S13, the mapping table searcher 22 reads 

out a mapping data item, associated with the pixel selected by 
the timing generator 25, from the mapping table MPT specified, 
and then outputs the data to the display mode changer 23. Re- 
sponsive to the state signal SI, the changer 23 rewrites, in 

25 Step S20 if necessary, the mapping data that has been read out 
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by the mapping table searcher 22 and then outputs the updated 
data to the pixel synthesizer 21. This Step S20 will be de- 
scribed in further detail later. 

Subsequently, in Step S30, in accordance with the mapping 
5 data input, the pixel synthesizer 21 synthesizes the pixel 
value(s) of the associated camera image(s) for the pixel se- 
lected by the timing generator 25 with the existent ones and 
then outputs the result to the video signal generator 24. 
Next, in Step S14, the video signal generator 24 transforms 
10 the input pixel values for the synthesized image into a video 
signal and outputs the signal to the display device 3 synchro- 
nously with the input of the timing signal from the timing 
generator 25 . 

Thereafter, the timing generator 25 determines in Step 
15 S15 whether or not the pixel that has just been processed is 
the last one of the given frame. If the answer is YES, then 
the current frame is regarded as having been processed suc- 
cessfully and processing of the next frame starts. Otherwise, 
the process advances to Step S16, in which the selection of 
20 the next pixel is instructed. Then, the image processing sec- 
tion 1 performs the series of processing steps S13, S20, S30 
and S14 all over again on the next pixel selected by the tim- 
ing generator 25 in Step S12. And this processing loop will 
be repeatedly carried out until the answer to the query of 
25 Step S15 becomes YES, i.e., until the last pixel of the cur- 
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rent frame is processed. 

In the illustrated embodiment , the processing is carried 
out on a f rame-by-f rame basis, but may naturally be performed 
on one field after another. 
5 FIG. 8 is a flowchart illustrating specifically how the 

display mode changer 23 operates in Step S20 shown in FIG. 7. 

First, in Step S21, the changer 23 checks the information 
given by the input state signal SI about the opened/closed 
state of some vehicle movable part (e.g., right door, left 
Ip 10 door, trunk or hood). If the movable part is open (i.e., if 
13 the answer to the query of Step S22 is YES), then the process 

fy advances to Step S23 to know the numbers of the cameras mount- 

!^ ed on that movable part by reference to the table (e.g., Table 

:ij 2) pre-stored on the changer 23. On the other hand, if the 

V2 15 movable part is closed (i.e., if the answer to the query of 
Step S22 is NO), then the process, jumps to Step S27 , in which 
the changer 23 determines whether or not all of the movable 
parts have been checked. If the answer is YES, then the proc- 
ess advances to the next step S28. Otherwise, the process re- 
20 turns to Step S21, in which the changer 23 checks the 
opened/closed state of another (or the other) movable part. 

Once the numbers of the cameras on the movable part in 
question are known in Step S23, the process advances to the 
next step S25. In Step S25, the changer 23 determines whether 
25 or not any of those camera numbers corresponds to one of the 
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camera numbers included in the mapping data. If the answer is 
YES, the process advances to the next Step S26, in which the 
mapping data about the camera with that number is rewritten to 
indicate that the camera is temporarily disabled. In the ex- 
5 ample illustrated in FIG. 8, the camera number and the X and Y 
coordinates are rewritten into "-1". On the other hand, if 
the answer to the query of Step S25 is NO, then the process 
jumps to Step S27, in which the changer 23 determines whether 
or not all of the movable parts have been checked. If the an- 

10 swer is YES, then the process advances to the next step S28. 
Otherwise, the process returns to Step S21, in which the 
changer 23 checks the opened/closed state of another (or the 
other) movable part. 

When all of the movable parts have been checked (i.e., 

15 when the answer to the query of Step S27 becomes YES), the 
rates of necessity of the mapping data are re-calculated in 
Step S28 to end the processing. 

Hereinafter, it will be described specifically how the 
display mode changer 23 operates in this Step S20 where the 

20 mapping data item MP2 shown in FIG. 6 is input to the changer 
23. In the following example, only the right door of a vehi- 
cle is supposed to be opened for illustrative purposes. 

First, the state signal SI tells the changer 23 in Steps 
S21 and S22 that the right door of the vehicle is open. Thus, 

25 by reference to Table 2 stored thereon, the changer 23 knows 
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in Step S23 that cameras Nos . 1 and 2 are mounted on the right 
door opened. 

Next, the changer 23 determines in Step S25 whether or 
not the camera number "1" or "2" corresponds to one of the 
5 camera numbers included in the mapping data item MP2 . In this 
case, the camera number "1" is included in the mapping data 
item MP2, so in the next step S26, the data stored about the 
camera No. 1 is rewritten into disabled ones. As for the cam- 
era number "2" on the other hand, that number is not included 
10 in the mapping data item MP2 and there is no need to rewrite 
the data about it. 

Since the other movable parts are all closed in the il- 
lustrated example, the answer to the query of the next step 
27 is YES. So the process advances to the next step S28, in 
15 which the rates of necessity are recalculated. In the illus- 
trated example, the disabled camera No. 1 has its necessity 
recalculated as "0" and the other camera No. 0 has its neces- 
s ity rewritten into " 1 " . 

FIG. 9 is a flowchart illustrating the pixel synthesiz- 
20 ing step S30 shown in FIG. 7. 

First, in Step S31, the synthesizer 21 initializes the 
value of a pixel to be synthesized into "0". 

Next, in Step S32, the synthesizer 21 reads out the cam- 
era number, X and Y coordinates and rate of necessity from the 
25 mapping data item associated with the pixel. If the camera 
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number read out indicates that the camera is now enabled (i.e., 
if the answer to the query of Step S33 is YES), then the syn- 
thesizer 21 reads out and retains in Step S34 the pixel value, 
defined by the X and Y coordinates , from the image stored on 
5 the frame memory for the enabled camera. On the other hand, 
if the camera number read out indicates that the camera is now 
disabled (i.e., if the answer to the query of Step S33 is NO), 
then the synthesizer 21 retains and outputs a predetermined 
pixel value indicating the disability (e.g., black pixel 

10 value) in Steps S35 and S39, respectively. 

Where the camera with that number is now enabled, the 
synthesizer 21 weights the pixel, which has been read out and 
retained in Step S34, by the rate of necessity and then adds 
the weighted value to the pixel value to be synthesized in 

15 Step S36. 

Then, in Step S37, the synthesizer 21 determines whether 
or not the given mapping data item includes any other camera 
number. If the answer is YES, then reading a next camera num- 
ber is instructed in Step S3 8 and the process returns to Step 

20 S32 to make the synthesizer 21 repeatedly perform the process- 
ing steps S32 through S36 until the answer to the query of 
Step S37 becomes NO. When it is found in Step S37 that there 
is no other camera number left in the mapping data item, the 
process advances to the next step S3 9, in which the synthe- 

25 sizer 21 outputs the pixel values to be synthesized. 
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By switching the display modes of the synthesized image 
in this manner, no unnatural image will be presented to the 
viewer even if any movable part of a vehicle has changed its 
state to make the synthesized image unnatural. 

5 

EMBODIMENT 2 

FIG. 10 is a block diagram illustrating a configuration 
for a monitoring system according to a second embodiment of 
the present invention. In FIG. 10, each component also shown 

10 in FIG. 1 is identified by the same reference numeral and the 
detailed description thereof will be omitted herein. 

The system shown in FIG. 10 is characterized in that a 
display mode changer 23A gets camera images selectively input 
from the imaging section 1 to a pixel synthesizer 21A respon- 

15 sive to the state signal SI, thereby switching the display 
modes of the synthesized image. That is to say, switches 31 
are provided between the output terminals of the imaging sec- 
tion 1 and the input terminals of the pixel synthesizer 21A. 
On sensing the opened state of any of the movable parts in ac- 

20 cordance with the information given by the state signal SI, 
the changer 23A identifies a disabled camera, not available 
for generating a synthesized image, by reference to the infor- 
mation stored thereon (e.g., Table 2). Then, the changer 23A 
turns associated one of the switches 31 OFF to prevent the im- 

25 age taken by the disabled camera from being input to the pixel 
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synthesizer 21A. 

In the second embodiment, the images are also synthesized 
using a mapping table as in the first embodiment. However, 
the second embodiment is different from the first embodiment 
5 in that the step S20 of rewriting the mapping data is omitted 
from the processing flow illustrated in FIG. 7. That is to 
say, since the input of an image taken by any disabled camera 
is stopped mechanically or electrically in the second embodi- 
ment, obviously there is no need to perform the step S20 of 
10 rewriting the mapping data in accordance with the state of a 
movable part. Consequently, Steps S3 3 and S35 included in the 
processing flow shown in FIG. 9 are also omitted in this em- 
bodiment . 

It should be noted that this second embodiment is easily 
15 applicable to an image processing section including no mapping 
tables . 

In the present invention, the display device 3 is typi- 
cally a liquid crystal display, but may be any other type of 
display device (e.g., plasma display). Optionally, according 

20 to the present invention, the display of a global positioning 
system (GPS) for vehicles, i.e., a so-called "car navigation 
system", may also be used as display device 3. Also, the cam- 
eras 11 included in the imaging section 1 are typically color 
or monochrome digital camcorders each including a solid-state 

25 imaging device like a CCD or CMOS device. Alternatively, the 
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cameras 11 may also be any other type of imagers. 

In the foregoing description, the present invention has 
been described as being applied to a situation where a change 
in the state of a movable part like a door or hood of a vehi- 
5 cle alters the position or direction of a camera mounted on 
that movable part and thereby makes a resultant synthesized 
image unnatural. However, there are other imaginable situa- 
tions. For example, if a movable part has changed its state, 
then the image of the movable part may occupy a different por- 

10 tion of the image taken by a camera mounted on a fixed part, 
thereby making the resultant synthesized image unnatural. Mo- 
re specifically, the image of an opened door may be uninten- 
tionally captured by the camera mounted on the fixed part, not 
the camera mounted on the movable part that opened. 

15 Even so, a table showing a correspondence between the 

movable parts and the associated cameras, which are not mount- 
ed on those movable parts but should be affected by any state 
change of the movable parts, has only to be prepared for the 
changer 23 or 23A instead of Table 2. Then, the same effects 

20 as those of the foregoing embodiments are also attainable by 
performing similar processing steps. 

It should be noted that the information representing the 
state change of a movable part does not have to be digital one 
(i.e., opened or closed) but may be analog one (e.g., how wide 

25 the part is opened). For example, the normal mode still may 
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be selected where a door is slightly open, but should be 
switched into the alert mode when the angle formed by the 
opened door with the body of the vehicle exceeds a predeter- 
mined threshold value. It should also be noted that the mov- 
5 able parts in the present invention are not limited to doors, 
hood and trunk but may include tires , antennas , movable rear- 
view mirrors and headlights. For example, where the movement 
of a tire affects the synthesized image, the display modes may 

g be switched in accordance with the steering angle formed by 

ji 10 the wheel on which the tire is fitted. 

□ Furthermore, in the present invention, the "vehicle'' 

U means any of various types of vehicles including normal- 

« e stroke-volume automobiles, light automobiles, trucks and 

Jf buses. Also, so long as the inventive concept is imple- 

~f 15 mentable, specially equipped vehicles including crane trucks 
and shovel cars may also be vehicles of the present invention. 

In the foregoing description, the monitoring system and 
image processor of the present invention are supposed to be 
applied to vehicles. However, the present invention is equal- 
20 ly applicable to any other types of moving objects including 
airplanes, boats and ships. Furthermore, cameras for the in- 
ventive monitoring system may be placed on a still object to 
be monitored, e.g., shops, residences and showrooms. 

Moreover, the positions and number of cameras to be 
25 mounted are not limited to the illustrated ones. 
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Furthermore , part or all of the functions of the inven- 
tive image processor may be implementable by either dedicated 
hardware or software programs. It is also possible to use a 
storage or transmission medium that stores thereon a program 
5 for getting the inventive image processing executed by a com- 
puter either partially or entirely. 

As described above, even if any change in the state of a 
vehicle movable part has made a synthesized image partially 
unnatural, that unnatural part can be removed from the synthe- 

10 sized image presented to the viewer in accordance with the 
present invention. In addition, by switching the display 
modes of the synthesized image appropriately, it is also pos- 
sible to let the user know quickly and timely that a movable 
part is currently in a state unfavorable for vehicle's run- 

15 ning, for example. 
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